Skip to content

[Variant] Add variant_get access as Variant - #9681

Merged
alamb merged 12 commits into
apache:mainfrom
sdf-jkl:variant_get-variant
Jun 23, 2026
Merged

[Variant] Add variant_get access as Variant#9681
alamb merged 12 commits into
apache:mainfrom
sdf-jkl:variant_get-variant

Conversation

@sdf-jkl

@sdf-jkl sdf-jkl commented Apr 9, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

Check issue

What changes are included in this PR?

  • Added support for variant_get access as Variant
  • Added unit tests

Are these changes tested?

Yes, added unit tests

Are there any user-facing changes?

Can now extract unshredded VariantArray with variant_get

@github-actions github-actions Bot added the parquet-variant parquet-variant* crates label Apr 9, 2026
@sdf-jkl
sdf-jkl marked this pull request as draft April 9, 2026 22:59
@sdf-jkl

sdf-jkl commented May 23, 2026

Copy link
Copy Markdown
Contributor Author

@scovich Please take a look when available!

@sdf-jkl
sdf-jkl marked this pull request as ready for review May 23, 2026 20:36

@scovich scovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks promising!

Comment thread parquet-variant-compute/src/variant_get.rs Outdated
Comment thread parquet-variant-compute/src/variant_get.rs Outdated
Co-authored-by: Ryan Johnson <scovich@users.noreply.github.com>
@scovich

scovich commented May 28, 2026

Copy link
Copy Markdown
Contributor

Looks like there's a merge conflict?

@sdf-jkl

sdf-jkl commented May 28, 2026

Copy link
Copy Markdown
Contributor Author

@scovich fixed

@sdf-jkl

sdf-jkl commented Jun 9, 2026

Copy link
Copy Markdown
Contributor Author

@scovich could you take a look again?

Comment thread parquet-variant-compute/src/variant_get.rs Outdated
Comment thread parquet-variant-compute/src/variant_get.rs Outdated
@@ -204,7 +204,23 @@ fn shredded_get_path(
// Helper that shreds a VariantArray to a specific type.
let shred_basic_variant =
|target: VariantArray, path: VariantPath<'_>, as_field: Option<&Field>| {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This helper just became a lot more complicated, could use some inline code comments explaining how this new requested_variant stuff impacts the control flow?

e.g. "if the caller requested variant, unshred to binary it before continuing; if they also provided a path to traverse, don't pass a type so that the requested field can be fetched directly."

But that opens a bunch of questions:

  1. Why unshred the whole thing if the user provided a non-empty path? Wouldn't it be better to any shredding as deeply as possible first? Or did the caller of shred_basic_variant already do that?
  2. This code seems to ignore any shredded variant schema the user might have provided. I get that we don't necessarily want to tackle variant-get with custom shredding in this PR, but we should at least block it with an error instead of silently returning something different than was requested?

@sdf-jkl sdf-jkl Jun 11, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. It does already do that.
  2. Hmm... It's [Variant] Support Shredded Objects in variant_get: access as Some(DataType::Struct) (nested shredding) #8153.

I could do 2 but I'm not sure about variant_get custom shredding.

I checked Spark/duckdb/dremio variant_get implementations and it seems like no one does output shredding as a part of variant_get which is basically:

variant_get(arr, path, Some(shredding_schema)) ≡ shred_variant(variant_get(arr, path), schema)

There's a https://github.com/datafusion-contrib/datafusion-variant Variant integration and it strives for parity with spark/databricks.

Custom shredding will, however, skip unshred-reshred round trip that'd come from doing:

shred_variant(variant_get(arr, path), schema)

That's really a #8153 discussion

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair! So we should just check for and block typed_value in the variant schema, to avoid any confusion.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done here cf7e545. Sorry, should've split the comments/change/nit into separate commits

Comment thread parquet-variant-compute/src/variant_get.rs Outdated
sdf-jkl and others added 2 commits June 10, 2026 15:43
Co-authored-by: Ryan Johnson <scovich@users.noreply.github.com>
Comment thread parquet-variant-compute/src/variant_get.rs Outdated
@sdf-jkl

sdf-jkl commented Jun 15, 2026

Copy link
Copy Markdown
Contributor Author

@scovich PTAL again.

@scovich scovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@sdf-jkl

sdf-jkl commented Jun 16, 2026

Copy link
Copy Markdown
Contributor Author

:shipit:

@sdf-jkl

sdf-jkl commented Jun 22, 2026

Copy link
Copy Markdown
Contributor Author

@alamb PTAL 😎

@alamb

alamb commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

I'll merge it on @scovich 's approval :) Any chance you can resolve the merge conflicts?

@alamb
alamb merged commit d2f1611 into apache:main Jun 23, 2026
17 checks passed
@alamb

alamb commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

gogogo

thanks again

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

parquet-variant parquet-variant* crates

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Variant] Support Shredded Objects in variant_get: Access as VARIANT (Unshredding)

3 participants